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ABSTRACT 


Visual perception modeling is generally weak for game artificial intelligence 
(AI) and computer generated forces (CGF), or agents, in computer games and military 
simulations. Several tricks and shortcuts are used in perceptual modeling. The 
results are, under certain conditions, unrealistic behaviors that negatively effect user 
immersion in games and call into question the validity of calculations in fine 
resolution military simulations. In games, the most common method of determining 
whether an agent can observe a target is by casting a line-of-sight ray from the 
observer to the target (or visa versa). The programmer determines where and how 
many points represent the observer and the target. For human targets the top of the 
head is often used. Using this method, a target is detected when the line segment 
extends from the observer to some predetermined point on the target without 
intersecting another object in the scene, including terrain. However, if an object, 
even a small one like a leaf, obstructs this single target point at which the line 
segment intersects the target, the target is considered concealed. For military 
simulations more complex models of visual perception exist (for example, the Army’s 
ACQUIRE model). However, inputs to this and similar models still yield 
questionable results for individual units because of the fixed values often used. For 
example, in the ACQUIRE model a fixed parameter is used for background contrast, 
and for the purposes of the calculation, a soldier that is on top of a hill and backlit has 
the same contrast with a soldier that is in a shaded, wooded area. This is obviously 
not the case in reality. By determining what the computer-generated agent sees using 
methods similar to that used to generate the human players’ screen view in 3-D 
virtual environments, we hope to present a method that can more accurately model 
human visual perception, to enhance the ability of entities to “hide in plain sight”. 
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I. 


INTRODUCTION 


A. PROBLEM AND THESIS STATEMENT 

Visual perception modeling for agents, that is, non-player characters (NPCs) in 
computer games and computer generated forces (CGF) in military simulations, is 
generally weak. This has a negative impact on immersion in games and military training 
simulations and calls into question the validity of analytical military simulations. 

We present a novel method that can more accurately model human visual 
perception by determining what the computer-generated agent sees using methods similar 
to those used to generate the human players’ screen view in 3-D virtual environments. 
Specifically we deal with the major problem of an entity hiding in plain sight, as 
described below. A demonstration program build on DeltaSD, http://delta3d.org/, an 
open source simulation engine being developed at the MOVES Institute, is also 
described. 

B. MOTIVATION 

Several tricks and shortcuts are used in perceptual modeling. The results are, in 
many instances, unrealistic behaviors that negatively effect user immersion in games and 
military training simulations, and call into question the validity of calculations in fine- 
resolution, analytical military simulations. 

In computer games and 3-D graphics-based military simulation, the most common 
method of determining whether an agent can perceive a target is by casting a line-of-sight 
ray from the observer to the target (or visa versa). The programmer determines the 
number of line-of-sight rays that are to be used and where these lines will be traced to on 
the target. For targets that represent humans, a single line projected to the top of the head 
is often used. Using this method, a target is detected when the line segment extends from 
the observer to some predetermined point on the target without intersecting another 
object in the scene, including terrain. However, if an object, even a small one like a leaf, 
obstructs this single target point at which the line segment intersects the target, the target 
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is considered eoneealed. One ean easily envision a similar situation for multiple line-of- 
sight rays. 

For military simulations more eomplex models of visual pereeption exist, 
including the Army’s ACQUIRE model. However, inputs to this and similar models still 
yield questionable results for individual units beeause of the fixed values are often used. 
For example, in the ACQUIRE model a fixed parameter is used for baekground eontrast 
and, for the purposes of the ealeulation, a soldier that is on top of a hill and backlit has 
the same contrast as a soldier that is in a shaded, wooded area. This is obviously not 
realistie. Though several efforts have been made to improve this shorteoming, for 
example see (Champion 99), pereeption situations are still eategorized for the type of 
landseape and not based on the speeilies of the eurrent situation. For example, a fixed 
eontrast is used for a European forest environment, but does not use the aetual eontrast 
between an individual soldier and the baekground. A different environment like a South- 
Ameriean jungle simply uses a different constant. 

1. Hiding in Plain Sight 

Consider the example of hiding in plain sight. A player might hide within the 
line-of-sight of the NPC, but use eamouflage, smoke, shadow, or elutter to eoneeal him or 
herself Using eurrent line-of-sight models this player would be deteeted without 
eonsideration given to his careful seleetion of a “hiding” plaee. This behavior is 
substantially different than the behavior that we would expeet from a human, whieh 
would be effeeted by faetors sueh as eamouflage. 


C. THESIS SCOPE 

This thesis provides a different methodology for providing inputs to a pereeption 
system and algorithms for using those inputs. Demonstration software that was 
developed, based on the Delta 3D open souree simulation engine, is also diseussed. 
Issues related to aeeelerating pereeptual ealeulations using graphies hardware are not 
eovered, though a related thesis along this line is discussed in Chapter II. 


2 



D. THESIS ORGANIZATION 

The remainder of this thesis is organized into 4 chapters as follows: 

1. Chapter II - Related Work 

Describes several works related to this thesis, including works in game and 
military simulation related domains. 

2. Chapter III - Algorithms 

Describes the algorithms that were developed for this thesis. Also, there is some 
discussion on how such algorithms might be implemented in more thorough simulations. 

3. Chapter IV - Demonstration Software 

Describes the demonstration software that was developed for this thesis. The 
demonstrations software provides a simple proof of concept for the algorithms and ideas 
presented in the thesis. 

4. Chapter V - Analysis and Conclusions 

The chapter provides some simple analysis and draws some conclusions for the 
thesis. Also, suggestions for future work and possible improvements are also provided. 
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II. RELATED WORK 


A. INTRODUCTION 

There has been a good deal of work done related to pereeption for intelligent 
software agents in virtual environments, and most of the more eomplex and important 
work has been done in the military simulation world. Pursel provides a great overview of 
many of the more important military algorithms and teehnologies. Mueh of the material 
on military simulation will be a summary of (Pursel04) that will be provided here for 
eontinuity. 

B. DETECTION MODELS IN MILITARY SIMULATIONS 

I. NVESD ACQUIRE, its Applications and Modifications 

Originally, the U.S. Army's Night Vision and Eleetronie Sensors Direetorate 
(NVESD) ACQUIRE model was developed to prediet target aequisition performanee for 
imaging systems operating in the visible, near infrared, and infrared spectral bands. 
ACQUIRE takes as inputs target characteristics, atmospheric conditions, and sensor 
operating characteristics to predict probabilities associated with a particular target 
acquisition task. Specifically, the probability of detecting the target given an infinite 
amount of time and the time-dependent probability of detection are calculated. 
ACQUIRE has two modes of operation; target spot detection and target discrimination. 
Target spot detection calculations are based on signal-to-noise ratio theory and require 
characterizing the system by either minimum detectable contrast or minimum detectable 
temperature difference. Target discrimination calculations are based on a two- 
dimensional Johnson cycle criteria methodology, (Johnson58), and require characterizing 
the system by either minimum resolvable contrast or minimum resolvable temperature 
difference, depending on if a visible or infrared result is of interest. The probabilities 
predicted by the model represent the expected performance of an ensemble of trained 
military observers with respect to an average target having a specified signature and size 
(NVESD 1). A more extensive description of the internal workings of ACQUIRE is 
deferred to the algorithms chapter of this thesis. 
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Recent military simulations that use the ACQUIRE model, to varying degrees, 
include the Combined Arms and Support Task Force Evaluation Model (CASTFOREM), 
the Janus Training Simulation, and the U. S. Marine’s Team Tactical Engagement 
System (TTES). See (Pursel04) for a more detailed description of how these simulations 
use ACQUIRE. 

Several extensions to the basic ACQUIRE model have been made. One of the 
more significant was by (Champion99). Champion quantifies the effects of vegetation on 
target acquisition in the context of the ACQUIRE model. 

2. Individual Combatants 

Reece and Wirthlin (Reece96) describe internal visual and audio models for 
individual combatant computer generated forces (CGFs), and some typical behaviors that 
result from these models. For the visual model an ACQUIRE-like model is used that was 
adapted from (Eind95). Reece and Wirthlin implement these models in the Team 
Tactical Engagement Simulator (TTES), a 3-dimensional simulation developed by the US 
Marine Corps using distributed interactive simulation technology. TTES, which later 
became the Small Unit Tactical Training (SUTT), is a virtual-reality training device that 
supplements live fire exercises for the Marine Corps. Training is conducted in a realistic 
combat environment where students, armed with simulated weapons, encounter 
computer-generated hostile and neutral characters. 

C. COMPUTER GAMES, VIRTUAL ENVIRONMENTS AND RELATED 

TECHNOLOGIES 

I. Synthetic Vision 

Blumberg describes a method where frames rendered from a synthetic creature's 
perspective are used as inputs to perception. Blumberg terms this approach "synthetic 
vision." Blumberg used input frames for creature steering in a virtual, indoor, 3- 
dimensional world in (Blumberg97) and (Blumberg96). Motion energy (Figure 1) is 
calculated for each half of the image and the virtual creature (in this case a dog) is able to 
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make course corrections and stay within the hallway boundaries, avoid obstacles, and 
approach items of interest in a complex “Doom”-like environment. 



Figure 1. Blumberg's Synthetic Vision 


2. Visibility and Concealment Algorithms 

Darken describes improved algorithms for determining the visibility of and 
finding concealment points for agents in 3D environments in (Darken04): 

Many military simulations and computer entertainment products share a 
need to model the ability of individual entities to see one another and to 
hide from one another in a 3D virtual environment. The traditional line- 
of-sight (LOS) visibility model can cause serious problems with hiding 
behavior. (Page 1) 

Algorithms of varying computational complexity include shadow volume, depth 
map, and sensor grid approaches. A more detailed description of the concealment 
algorithms as well as an implementation of the sensor grid approach can be found in 
(Morgan03). The sensor grid approach consists of laying a grid of sensors around the 
agent that wants to be concealed and then testing the visibility of those sensors from the 
agent(s) that it wishes to be concealed from. The recommended sensor pattern layout is 
illustrated in Figure 2. 
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Figure 2. Sensor grid eoneealment approaeh 



3. Using the Graphics Processing Unit (GPU) 

Pursel studied the feasibility of using the GPU for doing pereeption calculations 
in (Pursel04). Vertex and fragment shader programs are used to make comparisons of the 
stored images. All the renders and calculations are performed on the Graphics Processing 
Unit (GPU) and the result is returned to the agent in reduced form for decision-making. 
Pursel showed that such calculations can be done on the GPU and, thus, avoid the costs 
associated with returning large amounts of data from the GPU to main memory. 
Presently, a good deal of vendor specific code is required to accomplish these 
calculations on the GPU. Future technologies will likely increase the ease of doing these 
calculations on the GPU or the cost of returning the full-sized frames to the CPU for 
calculation. 
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III. TARGET DETECTION ALGORITHMS 


A. INTRODUCTION 

In this chapter we deseribe the algorithms that were developed to more aecurately 
model human visual pereeption. The Army's Night Vision and Eleetronie Sensors 
Direetorate (NVESD) ACQUIRE model was ehosen as the basis for our visual pereeptual 
modeling. This was done for several reasons. Eirst, ACQUIRE is a model that is often 
used for modeling visual pereeption in military simulations. (NVESD 1) states that “the 
ACQUIRE model ean be applied to performanee predietion of Direet View Optieal 
(DVO) sensors [ineluding the naked eye] but is not reeommended for that purpose.” In 
praetice it has been used for DVO often (see Chapter II). Also, ACQUIRE is a well- 
known and widely used model. However, in most implementations many of the 
important input parameters in ACQUIRE are estimated with fixed values. Our 
implementation uses speeific situational data, taken from a seene render, to a greater 
extent then has been done previously. 

The use of ACQUIRE with the improved input parameters has lead to a 
perceptual model superior to line-of-sight. However, the basie ACQUIRE algorithm still 
performs poorly in some situations. These situations will be eovered in greater detail 
below. In this ehapter, we present an extension to the basie algorithm that uses by-eolor 
eontrast and eolor histograms that improve the algorithm performanee in many of these 
situations. 

B. GATHERING DATA FOR COMPARISON, THE USE OF FALSE 

COLORING 

The first step in our algorithm requires the generation of two renders from the 
perspeetive of the observing agent: one normal render and one render with any of the 
targets of interest rendered using flat, false eoloring. By false eoloring we mean that any 
targets of interest are rendered in a fiat, pre-determined eolor value that will indieate that 
it is a subjeet of the pereeption. In our demonstration program red is used, that is the 
RGB (Red, Green, and Blue) values are set to 1.0, 0.0, and 0.0 respeetively. The effect is 
demonstrated in Eigure 3. 
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Figure 3. Figure rendered with false eoloring 


In gaming and training applieations, measures will be taken to keep this render 
from going to the sereen. There are several methods for doing this, ineluding rendering 
to texture. However, for the sake of eode and software exeeution elarity, this off-line 
rendering was not implemented in the demonstration program. Onee the two images are 
generated they are further proeessed as follows: 

1. The Line of Sight Check 

At this point we ean easily eheek for line of sight. If the false eolor render 
produeed any pixels with false eoloring we know that line-of-sight exists. If no false- 
eolor pixels were rendered, line-of-sight does not exist and we ean exit with no 
probability of deteetion. 
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2. The Mini-Render 

Next we reduce the size of the images to include just the target and the area 
surrounding the target. We call this the mini-render, Figure 4. 


Figure 4. A Mini-Render 

There are several reasons for doing this. First and foremost, the area directly 
surrounding the target has the greatest bearing on how well hidden the target is. The 
algorithms that are used compare the target and non-target pixels values. Often pixels 
that represent the target are only a small number of the pixels in a given render. If all the 
non-target pixels in the scene are used one might obtain average values that are not 
indicative of the area surrounding the target. 

For example, imagine a figure in camouflage hidden in a heavily shadowed forest 
with a washed out hazy sky above. In this situation the hazy sky would greatly effect the 
pixel values for the background, but, in reality, would have little bearing on whether the 
target stands out in the scene or not. Another, example is a figure silhouetted in a 
doorway at night. If the scene were taken in its entirety it is possible that we would 
calculate very little contrast between the figure and the background. However, given that 
the figure is backlit in a doorway any observer would easy detect him/her. 

C. THE ACQUIRE-LIKE ALGORITHM 

ACQUIRE was used as the basic algorithm for calculating detection for the 

reasons stated above. Our algorithm, as implemented, provides the following: the 

probability of detection of the target given an infinite amount of time (i.e. the time- 

independent detection probability) and the time-dependent probability of detection (i.e. 

the probability that a target will be detected before a given number of seconds). Our 
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algorithm combines the two operating modes of ACQUIRE; target spot detection and 
target discrimination. 

The first input to ACQUIRE is the target signature. Eor visual systems this 
consists of the contrast difference between the background and the target. ACQUIRE 
starts with the inherent contrast, that is, the eontrast with no degradation for atmospheric 
or combat-induced eonditions, like smoke, fog, refleetanee, ete. However, in the virtual 
environment these factors are represented graphieally; thus we ealculate the eontrast 
direetly and do not need to estimate the effeet of these obseurants using the ACQUIRE 
empirieal equations. 

To caleulate the contrast for our basic ACQUIRE implementation we take the 
weighted average of the pixels in the mini-render that are part of the target, pt, and that 
are part of the target background, pb. However, ACQUIRE defines the contrast by the 
irradiance of the target and baekground in units of watts per meters square (W/m ). The 
units that we have in the virtual environment are pixel values. As it turns out there is a 
relationship between the units that AQUIRE uses and the pixel values we have, though 
the relationship is not a linear one. 

Watts per meter squared is proportional to lumens as follows: 

lumen = -^Watts@555nm (1) 

683 


Also, lumens are related to Eux as follows: 

, lumen 
lux = ;— 


( 2 ) 


m 


If one ignores the color dependence inherent in lumens we ean eonelude that Eux 
is proportional to Watts per meter squared; 


Lux oc 


watts 


m 


( 3 ) 


There is a known relationship between Eux and pixel value. It is a squared 
relationship and thus. 


12 



watts 


(4) 


m 


■oc pixel_value^ 


pixle value is numeric value a given pixel. Given the results of these 
calculations, we now are able to use pixel values to calculate the contrast. Thus, we are 
able to substitute the pixel value squared in for the irradiances of the target and 
background. Calculation of contrast becomes: 


C = 



( 5 ) 


C is the contrast, pt is the average pixel value of the target and pb is the average 
pixel value of the background. To calculate the p values we average the R, G, and B 
values for each pixel over either the target or background, as appropriate, within the mini¬ 
render. 

Next we need some measure of the angle, 0, which the angle that the target 
occupies in the field of vision of the observer. Figure 5. 


Figure 5. Relationship between 0 and the number of pixels used to represent the 

target 

In ACQUIRE this is estimated with the ratio of the critical dimension (CD), i.e. 
the square root of the exposed surface area of the target, and the range. This yields an 
approximation of 0. We estimate this parameter to within a constant factor using the 
square root of the number of target pixels that can be seen by the observer, 
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num_of_pixels, a piece of information that is readily available in our graphical 
environment. 



where E = 2.1 + 0.1 {N / A^50) (9) 

We use a default value of 1.0 for N50, as suggested in (Lind95) and (Reece96). 
N50 is the number of cycles required to be resolved on the target in order to achieve a 
50% probability of discrimination (NVESDl). In other words, N50 is the number 
necessary for half of a group of trained observers to detect a given target, on average, 
under certain lighting conditions. We believe that a better estimation for N50 can be 
derived from experimentation, but such is beyond the scope of this study. 

We further calculate the time dependent target detection as 

( 10 ) 

Where Pdet is the time-independent probability of detection calculated above, t is 
the length of time of interest, m is the number of sensor fields in the field of regard (for 
our software we assume a value of 1) and x is the mean detection time in seconds for all 
observers that eventually detect the target. The field of regard (FOR) is the size of the 
area that an observer is responsible for scanning. The FOR can be a few degrees up to 
360 degrees. Experimental data indicates that the value of x is related to the overall 
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difficulty of detecting the target (NVESDl). ACQUIRE uses an approximation for x, 
which we adopt, that is given by 


6.8 

r =-see 

N/N50 


( 11 ) 


The result produced by our algorithm solves several of the problems with the 
simple line-of-sight ealeulations used in many applications as a substitute for pereeption. 
Eor example, it easy takes into account fog. Eog has the effect of dimming the eontrast 
between a target and the background. In dense fog there may be no contrast at all. 
Eigure 6 depicts this situation. Eigure 7 shows the same scene, but with the target in false 
coloring. 



Eigure 6. Eoggy Scene with line-of-sight to the target, but the target is undetectable 
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Figure 7. False eolor version of Figure 6 


If line-of-sight were used in this situation the target would have been deteeted 
beeause line-of-sight does exist. Clearly, however, the target should not be visually 
deteeted. 

One weakness of the basic ACQUIRE algorithm is that we use pixel values for 
the target and background are averaged over all the colors. Because of this there is the 
possibility of an error given the correct situation. Imagine a figure wearing a red suit in a 
green forest. Obviously, any non-color blind observer would spot this anomaly rather 
quickly. However, if the red color of the suit and average green color of the forest are 
equal (assume the other colors are the same), they will average out to the same value and 
the algorithm would notice no contrast between the target and the background and, thus, 
return a zero probability of detection. Obviously, this is incorrect. Modifications to the 
above algorithm that alleviate this problem are presented below. 
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D. MODIFICATIONS TO THE ACQUIRE-LIKE ALGORITHM 


I. The By-Color ACQUIRE-Like Algorithm 

Rather than taking the average pixel value for the target and the background, we 
compute contrast for each color (red, green, and blue) and calculate the overall contrast as 
follows: 


2 2 

1 

J_ 

2 2 

1 

j_ 

2 2 

Pt_red Pb_red 

Pt green Pb green 

P t blue Pb blue 

Pb red 

1 

Pb green 

1 

Pbhlue 


We give an equal weighting to each of the three component colors. Equation (12) 
solves the problem of the red target in a green forest presented above. 

2. Color Histograms 

We have also implemented a version of the calculations that uses color histograms 
to compute the contrast. There is one histogram for the target and one for the background 
and for each color: red, blue and green. This yields a total of 6 histograms. Our 
implemented version uses 8 color bins, which was an arbitrary choice. We then 
innumerate through the mini-render and count the number of pixels within each of the 
pixel intensity ranges, according to Table 1. 
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Bin Index 

Minimum Pixel Intensity 

Maximum Pixel Intensity 

0 

0 

31 

1 

32 

63 

2 

64 

95 

3 

96 

127 

4 

128 

159 

5 

160 

191 

6 

192 

223 

7 

224 

255 


Table 1. Pixel value ranges that eause eounts within eertain bins 


Figure 8 illustrates a histogram for the red pixel values of a target and Figure 9 
illustrates the histogram of the red pixel values of the baekground. Note that the number 
of pixels will vary between the target and the baekground. 
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Red Target Histogram 



Bin 


Figure 8. Flistogram for the Red Pixel Values of a Target 


Red Background Histogram 



Bin 

Figure 9. Histogram for the Red Pixel Value of the Background 

Once the pixel iteration is complete the histograms are converted to probability 
mass functions (PMFs), that is, the sum of all the bins within the histogram are equal to 
one. We then calculate the LI difference, i.e. the sum of the differences between the 
background and target for each bin. Figure 10. 
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Figure 10. The sum of the differences between the target and background counts 
within each bin are used to calculate the LI difference 


Equation (13) is used to calculate the overall contrast using the PMFs given all 6 
histograms. 

C = — ^ I trgtR. - bckR. | + — ^ | trgtG- - bckG -1 — ^ | trgtB^ - bckB. | (13) 

^ bins ^ bins ^ bins 

The result of the contrast calculation is used as the contrast input to the remaining 
calculations above. 
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IV. DEMONSTRATION SOFTWARE 


A. INTRODUCTION 

This chapter gives an overview of the demonstration software developed in 
conjunction with this thesis. Our software is built with the Delta 3D open-souree 
simulation engine, whieh is actively under development at the MOVES Institute at the 
Naval Postgraduate School in Monterey, CA. The goal of the demonstration software is 
twofold; First, the software provides a proof of concept for the basic idea put forth in this 
thesis, i.e., renders from a given perspective can be used as inputs for perception 
algorithms. Additionally, our software was used in this thesis and ean be used in the 
future as a test bed for the development and testing of pereeption algorithms. 



Figure 11. A scene from the demonstration software 


Figure 11 portrays a seene in a town with one human-like figure. In our software 
the human figure acts as the target for the algorithms. 
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B. FEATURES 


All the features in the software are eontrolled with keyboard input. The user ean 
eontrol several aspeets of the seene using the keyboard ineluding the position and 
orientation of the figure, the amount of visibility into the fog (when on) and the time of 
day. Additionally, the user ean initiate the exeeution of the algorithms at any time. 

1. Figure Position and Orientation Control 

Figure position and orientation are ehanged using the left and right arrow keys 
and the “W” key. “W” eauses the figure to walk forward. The eombination of Shift and 
“W” eause the figure to run forward. The right arrow key eauses the figure to turn right 
and the left arrow key eauses the figure to turn left. 

2. Fog Control 

Visibility through the fog within the seene is eontrolled using the plus (+) and 
minus (-) keys. Pressing the plus key inereases the visibility in the seene. If visibility 
reaehes beyond 100 the fog is turned eompletely off (until the minus key is pressed 
again). The minimum visibility is 1. Visibility ehanges in inerements of 10. 


3. Time of Day Control 

Time of day is eontrolled using the number pad. Pressing a given number will 
ehange the time of day within the seene as follows: 

0 0200 

1 0400 

2 0600 

3 0800 

4 1000 

5 1200 

6 1400 

7 1600 

8 1800 
9 2000 

Table 2. The number key and eorresponding time of day eombinations in our 
demonstration software 
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4. Executing Perception Calculations 

Perception calculations are executed by pressing the “S” key. Output is sent to 
the command-line console. The user must ensure the figure is stationary during percept 
calculations. Code was not included to freeze the scene during screen captures. If the 
figure is moving during a capture the false color and normal color frames will not 
coincide and any calculations will not be accurate. 


C. IMPLEMENTATION 

The goal of this section is to give a brief overview of the class structure of the 
demonstration program. The demonstration program is implemented using the Delta 3D 
open source gaming and simulation engine (DeltaSD). Delta 3D is written in the C++ 
programming language and developed in Microsoft Development Environment, Version 
7.1. Delta 3D includes many easy to use classes to make simulation development easier. 
These classes were used or extended whenever possible. Additional open source code 
was used and credit is given below. 

I. Selected Classes and Methods 

The program uses the following classes and methods. A brief discussion of each 
is provided: 

Demo class . This is a class derived from the Delta 3D Application class. It is the 
base level class for the application and it contains the basic components required for the 
application including the application window, scene camera, and simulation loop. This 
class is responsible for loading objects into the scene as well as application flow control. 
It also allows for alterations to or calculations on the scene to occur at a specific point in 
the render process, i.e., in the post-render phase (but before the next render). Lastly, the 
control keys for the program are mapped in this class. 

SyntheticPerception class . This class is responsible for doing all the perception 
calculations. The methods that produce the probability of detection calculations take 2 
images as arguments: one normal image and one image with a false colored target. The 
calculations described in chapter 3 are implemented in this class. 
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CharController class . Controls target movement in the software. Also eontrols 
the false eoloring of the target. 

Image elass . This is a utility elass that takes eare of many of the details assoeiated 
with image aeeess. Onee sereens are read from the frame buffer, an Image objeet is 
generated from the eharaeter array. This elass was written by Chris Darken. 

Bitmap utilities . This is a set of bitmap utilities written by Miehael Sweet. It 
allows OpenGL-eompatible images to be written to and from BMP (bitmap) files. 
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V. ANALYSIS, CONCLUSIONS, AND SUGGESTED EUTURE 

WORK 


A. INTRODUCTION 

Each of the perception algorithms discussed in this thesis has some advantages 
and disadvantages, which we will diseuss in more detail in this section. Also, we provide 
some eursory eomparisons between the outputs of the algorithms versus some 
experimental data that was gathered during a pilot study. Finally, we will provide some 
eonclusions, insights, and suggestions for future work. 

B. ANALYSIS OF ALGORITHMS 

I. Line-of-Sight Algorithms and Graphics Pipeline Issues 

Many of the disadvantages and shortcomings of line-of-sight algorithms have 
been previously described in this thesis. These included the inability to deal with fog and 
camouflage, i.e., situations were there is very little contrast between the target and the 
background. Additionally, problems may oecur if only a small part of the target is 
obstructed by an obstacle, yet it may include that point to which the line-of-sight rays are 
traced. In other words, it runs the risk of not deteeting a mostly visible target. 

However, there are some advantages to using line-of-sight calculations. First and 
foremost it can be fast, especially when the number of line-of-sight calculations is small. 
Also, it holds an advantage over the paradigm presented in this thesis in that no data must 
be read back from the graphies pipeline. With current hardware this is a very costly 
procedure. In effect, all traffic on the graphics pipeline must be stopped to allow for the 
return data, which can be substantial. 

There are two possible solutions to this limitation. The first solution was 
presented in (Pursel04). Pursel demonstrated that perception caleulations can be 
executed on the Graphies Processing Unit (GPU) and with only a result being returned to 
main memory, thus alleviating the need to return large chunks of data. General Purpose 
Computing on the GPU is an active area of interest and research because of the ever- 
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increasing speed and highly parallel nature of the GPU. The GP2 eonferenee (GP2_04) 
is a forum for sueh researeh and diseussion. However, this approaeh also has its 
problems; First, many of the meehanisms for earrying out sueh ealeulations, i.e., 
Applieation Programmer Interfaees (APIs), are vendor speeifie. Also, most of these APIs 
are not well doeumented. Some standards have been developed reeently, but there is still 
no elear single standard API for GPU programming. Mueh of the programming 
eommunity seems to be waiting for a single standard before venturing onto the GPU. 

The other possible solution to the graphies pipeline issue is the ever inereasing 
bandwidth from the graphies pipeline baek to main memory. PCI Express, the new 
input/output bus standard, promises preeisely this inerease. In the near future, reading 
baek pixel data will likely be a less bus-intensive proeedure. 

2. Determining Line-of-Sight Using False Color 

Determining target line-of-sight using false eolor is a trivial matter: If there are 
any false-eolor pixels in the seene, line-of-sight exists. In faet, an even greater amount of 
information is easily available. For example, if the seene is rendered with nothing but the 
target (no buildings, ete) one ean easily determine preeisely the pereentage of the target 
that is exposed, not just the estimate provided by multiple line-of-sight rays. Of eourse, 
eaution must be taken when using false eolor to ensure that the designated false eolor is 
not used anywhere else in the seene. If this is the ease, any ealeulations would be 
invalidated. 


3. The ACQUIRE-Like Algorithm 

The ACQUIRE-like algorithm introduees several advantages over line-of-sight 
rays and line-of-sight using false eoloring. The most obvious example is its ability to 
deal with fog. In fog, a elear line-of-sight may exist, but the target may still be 
eompletely undeteetable. The ACQUIRE algorithm deals with this by eomparing the 
target eontrast with the baekground eontrast. Essentially, if the average eontrast of the 
target and baekground are similar then a target is diffieult to deteet. Intuitively this 
makes perfeet sense. However, this is not a solution for all oases (see seotion B below). 
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ACQUIRE does not deal with issues like elutter, texture and eontinuity. These are all 
faetors that effeet deteetion likelihood in a human. Also, the averaging of eolors that 
ACQUIRE requires ean have some undesirable eonsequenees (as pointed out at the end 
of Chapter 3). 

4. Improving the ACQUIRE-Like Algorithm Using Color Components 
and Color Histograms 

We dealt with the problem of averaging eolors in two ways: averaging eaeh eolor 
individually (then taking the weighted average) and using eolor histograms. These 
methods solve the averaging problem, but still leave unresolved the other issues of 
elutter, texture, ete. 

C. EXPERIMENTAL DATA AND RESULTS 

In this seetion we provide some eomparison and eomments between the 
algorithms presented in this thesis and a pilot pereeption experiment eondueted at the 
Naval Postgraduate Sehool. 

I. The Pilot Experiment 

We eondueted a target deteetion pilot experiment at the Naval Postgraduate 
Sehool (NPS) in whieh fourteen subjeets were seated in front a large sereen display and 
statie images were displayed to them as shown in Eigure 9. All subjeets were students at 
NPS and had varying military baekgrounds and experienees. 
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Figure 12. A subject receiving instructions for the pilot experiment 

A total of 37 images were displayed, the first 4 being example images that were 
used to indoctrinate subjects into the detection task. For the 33 remaining screens 
subjects were asked to search the screen for a single human-like target. Subjects would 
indicate that they had detected the target by left clicking on the target using a mouse. 
Subjects were asked to find the target as quickly as possible, but to take all the time they 
needed. A small number of the screens had no target and the subjects could indicate that 
they perceived this by performing a different action. Each user was given the same 
scripted instructions at the beginning of the experiment. Also, users filled out 
questionnaires after the experiment that disclosed what level of experience they had with 
first-person shooter games, in general, and the games used for the screen shots, 
specifically. They had a wide range of experiences. There did not appear to be any 
correlation between game experience and detection times. 

2. Selected Screen Shots: Experimental vs. Algorithmic Results 

Flere we present a sampling of the images used in the pilot experiment, the results 

from those experiments, and the results from processing by our algorithms. This will 

both highlight the strengths and weaknesses of the approach we have taken and argue for 

further research in the area of perceptual modeling. It should be noted that ACQUIRE is 
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relatively well adapted to the types of seenes presented in our data set. The issues 
regarding averaging eolor are not represented here. Only the images with the more 
interesting results are ineluded. 
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Screen Shot 06 



Figure 13. Screen Shot 06 



Figure 14. Screen Shot 06 with false coloring 
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Figure 13 depicts Screen Shot 06, as displayed to subjects in the pilot experiment. 
Figure 14 shows the same screen with the target in false coloring (to aid in target 
detection by the reader). The false coloring of the targets in these images was done by 
hand using Adobe Photoshop. The results of the pilot experiment for this image were as 
follows; 

• Of the 14 subjects 8 were able to detect the target in this scene, a 
0.5714 probability of detection 

• Of the 8 that successfully detected the target 4 did so within 5 
seconds, thus a 5-second, time-dependent probability of detection 
of 0.2857. Detection times were distributed as per Figure 15 
below; 


Time to sucessful detection 



Seconds to Detection 


Figure 15. Distribution of successful experimental detection times for Screen Shot 06 

Results from executing the perception algorithms on Screen Shot06 yielded the 
following results; 

• The probability of detection for the ACQUIRE-like algorithm was 
0.6545, with a 0.3832 probability of detecting the target within 5 
seconds. 

• The probability of detection for the Color modified ACQUIRE-like 
algorithm was 0.6577, with a 0.3860 probability of detecting the 
target within 5 seconds. 

• The probability of detection for the Color histogram algorithm was 
0.5746, with a 0.3170 probability of detecting the target within 5 
seconds. 
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Table 3 below is provided for easier data eomparison; 


Souroe 

T ime-independent 

Probability of Deteetion 

Probability of Deteetion 
within 5 Seeonds 

Experiment 

0.5714 

0.2857 

ACQUIRE-like algorithm 

0.6545 

0.3832 

By-Color ACQUIRE 

0.6577 

0.3860 

Color Histograms 

0.5746 

0.3170 


Processing Frames . 








Using the ACQUIRE-like algorithm 






Ave. Target 

Fix Val 

: 42.8177 

Ave. 

Bckgrd 

Pix Val: 

44.8614 


Contrast Value: 

0.0890382 





n: 


1.19789 






e : 


3.53852 






pdet: 0.654507 
p(5.0): 0.383246 








Using the by-color 

ACQUIRE- 

like algorithm 





Ave. Target 

Red Val 

44.4309 

Ave. 

Bckgrd 

Red Val: 

45.8427 


Ave. Target 

Grn Val 

43.1492 

Ave. 

Bckgrd 

Grn Val: 

45.1685 


Ave. Target 

Blu Val 

40.8729 

Ave. 

Bckgrd 

Blu Val: 

43.573 


Contrast Value: 

0.0893846 





n: 


1.20255 






e : 


3.54178 






pdet: 0.657743 
p(5.0): 0.386073 








Using color histograms 







Target Red: [0 

178 

3 

0 

0 

0 

0 

0] 

Bkgrnd Red: [0 

80 

9 

0 

0 

0 

0 

0] 

Target Green: [2 

177 

2 

0 

0 

0 

0 

0] 

Bkgrnd Green: [0 

80 

9 

0 

0 

0 

0 

0] 

Target Blue: [7 

173 

1 

0 

0 

0 

0 

0] 

Bkgrnd Blue: [0 

80 

9 

0 

0 

0 

0 

0] 

LI diff: 


0.0810665 





n: 


1.09064 






e : 


3.46345 






pdet: 0.574564 
p(5.0): 0.316896 









Table 3. Deteetion probability eomparison table and program output for Sereen 
Shot 06 


We believe that the algorithm yields reasonable results for Sereen Shot 06, as this 
Sereen Shot was diffieult for the experimental subjeets beeause of the low level of 
eontrast between the target and the baekground. Contrast is one of the main inputs for 
the ACQUIRE model, and, our ACQUIRE-like algorithm was able to handle it in a 
reasonable manner. For example, eonsider a line-of-sight ealeulation on this target. The 
target would immediately be deteeted in most oases. However, almost half of the 
experimental subjeets were unable to deteot the target. Thus, in this ease our ACQUIRE- 
like algorithm displays a desirable, human-like behavior that a normal line-of-sight 
ealeulation would not. 
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Screen Shot 17 



Figure 16. Screen Shot 17 



Figure 17. Screen Shot 17 with false coloring 
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Time to sucessful detection 



Seconds to Detection 


Figure 18. Distribution of successful experimental detection times for Screen Shot 17 


Source 

Time-independent Probability 
of Detection 

Probability of Detection 

within 5 Seconds 

Experiment 

1.0 

0.9286 

ACQUIRE-like algorithm 

0.9998 

0.9470 

Color-modified ACQUIRE 

0.9998 

0.9470 

Color Histograms 

1.0 

0.9472 


Processing Frames . . . 


Using the ACQUIRE-like algorithm 

Ave. Target Pix Val: 112.84 Ave. Bckgrd Pix Val: 164.732 
Contrast Value: 0.530783 

n: 4.47246 

e: 5.83072 

pdet: 0.999839 
p (5.0) : 0.947044 


Using the color-aware ACQUIRE-like algorithm 

Ave. Target Red Val: 113.197 Ave. Bckgrd Red Val: 164.761 

Ave. Target Grn Val: 113.254 Ave. Bckgrd Grn Val: 164.761 

Ave. Target Blu Val: 112.07 Ave. Bckgrd Blu Val: 164.674 

Contrast Value: 0.530774 

n: 4.47238 

e: 5.83067 

pdet: 0.999839 
p (5.0) : 0.947044 


Using color histograms 


Target Red: 

[0 

0 

0 

70 

0 

1 

0 

0] 

Bkgrnd Red: 

[0 

0 

0 

0 

5 

41 

0 

0] 

Target Green: 

[0 

0 

0 

70 

0 

1 

0 

0] 

Bkgrnd Green: 

[0 

0 

0 

0 

5 

41 

0 

0] 

Target Blue: 

[0 

0 

0 

70 

0 

1 

0 

0] 

Bkgrnd Blue: 

[0 

0 

0 

0 

5 

41 

0 

0] 


LI diff: 0.887324 

n: 7.47672 

e: 7.93371 


pdet: 1 

p(5.0): 0.947196 


Table 4. Detection probability comparison table and program output for Screen 
Shot 17 


34 



































For Screen Shot 17, depicted in Figure 16, we onee again obtain a reasonable 
result. However, this shot has different properties from the previous sereen. Here we 
find that the target is nieely silhouetted against the baekground, though some attenuation 
due to haze is depicted. Additionally, the target is at a substantially greater distance from 
the observer. In this ease the contrast difference was substantial and negated the small 
number of pixels representing the target. 

From a human pereeption perspective there is another possible reason for the high 
deteetion rate. The eontrast is important, but so is the faet that the target is situated on 
top of a hill that is otherwise smooth. That being said, this is not a fact that is easily 
quantifiable for input to a perception algorithm. Some teehniques do exist, such as those 
in the field of computer vision, as in (ForsythOS). Using some of these techniques has the 
potential to further improve perception algorithms. 
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Screen Shot 18 



Figure 19. Screen Shot 18 



Figure 20. Screen Shot 18 with false coloring 
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Source 

T ime-independent 

Probability of Detection 

Probability of Detection 
within 5 Seconds 

Experiment 

0.2143 

0.0000 

ACQUIRE-like algorithm 

0.9990 

0.9323 

Color-modified ACQUIRE 

0.9990 

0.9324 

Color Histograms 

0.9997 

0.9469 


Processing Frames . . . 

Using the ACQUIRE-like algorithm 

Ave. Target Pix Val: 72.9273 Ave. Bckgrd Pix Val: 102.779 
Contrast Value: 0.496533 

n: 3.68239 

e: 5.27767 

pdet: 0.998973 
p(5.0): 0.932347 

Using the color-aware ACQUIRE-like algorithm 


Ave. Target 

Red Val: 

74.1455 

Ave. 

Bckgrd 

Red Val: 

104.357 


Ave. Target 

Grn Val: 

73.5818 

Ave. 

Bckgrd 

Grn Val: 

103.411 


Ave. Target 

Blu Val: 

71.0545 

Ave. 

Bckgrd 

Blu Val: 

100 569 


Contrast Value: 

0.496572 






n: 


3.68268 






e : 


5.27787 






pdet: 0.998973 
p(5.0): 0.932362 








Using color histograms 







Target Red: [0 

1 

54 

0 

0 

0 

0 

0] 

Bkgrnd Red: [0 

2 

120 

174 

57 

0 

0 

0] 

Target Green: [0 

3 

52 

0 

0 

0 

0 

0] 

Bkgrnd Green: [0 

4 

122 

176 

51 

0 

0 

0] 

Target Blue: [0 

8 

47 

0 

0 

0 

0 

0] 

Bkgrnd Blue: [0 

6 

130 

202 

15 

0 

0 

0] 

LI diff: 


0.573654 






n: 


4.25433 






e : 


5.67803 






pdet: 0.999731 
p(5.0): 0.946942 









Table 5. Detection probability comparison table and program output for Screen 
Shot 18 


The results from our algorithms versus the experimental data for Screen Shot 18, 
depicted in Figure 19, are less encouraging. Recall that ACQUIRE bases its detection 
probabilities on two main inputs: The target vs. background contrast and the size of the 
target within the visual field of the observer. For Screen Shot 18, the contrast between 
the target and the background is significant. It appears that many of the experimental 
subjects had problems with Screen Shot 18. Two reasons for this are apparent: (1) the 
target is in clutter, and (2) the target's body position is such that it is inline with the 
clutter. In effect, the target looks like it is part of the tree. ACQUIRE does not deal with 
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either of these types of concealment, nor do either of our modified algorithms. Again we 
suggest that the field of computer vision may have the solution to this non-trivial 
problem. Specifically, calculations that deal with texture would be helpful in this case. 
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Screen Shot 25 



Figure 21. Screen Shot 25 



Figure 22. Screen Shot 25 with false coloring 
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Time to sucessful detection 



4 5 6 8 9 10 More 

Seconds to Detection 


Figure 23. Distribution of successful experimental detection times for Screen Shot 25 


Source 

Time-independent Probability 
of Detection 

Probability of Detection 

within 5 Seconds 

Experiment 

0.9286 

0.4286 

ACQUIRE-like algorithm 

0.5829 

0.3234 

Color-modified ACQUIRE 

0.5796 

0.3209 

Color Histograms 

0.1845 

0.0678 


Processing Frames . . . 







Using the ACQUIRE-like algorithm 






Ave. Target Pix Val: 

38.466 

Ave. 

Bckgrd 

Pix Val: 

39.72 


Contrast Value: 

0.0621438 





n: 

1.10119 






e: 

3.47083 






pdet: 0.582869 
p(5.0): 0.323497 







Using the color-aware ACQUIRE-like algorithm 




Ave. Target Red Val: 

39.4204 

Ave. 

Bckgrd 

Red Val: 

40.2182 


Ave. Target Grn Val: 

38.6529 

Ave. 

Bckgrd 

Grn Val: 

40.4909 


Ave. Target Blu Val: 

37.3248 

Ave. 

Bckgrd 

Blu Val: 

38.4509 


Contrast Value: 

0.0619072 





n: 

1.097 






e: 

3.4679 






pdet: 0.57958 
p(5.0): 0.320875 







Using color histograms 
Target Red: [13 298 

3 

0 

0 

0 

0 

0] 

Bkgrnd Red: [18 251 

6 

0 

0 

0 

0 

0] 

Target Green: [16 297 

1 

0 

0 

0 

0 

0] 

Bkgrnd Green: [17 252 

6 

0 

0 

0 

0 

0] 

Target Blue: [24 289 

1 

0 

0 

0 

0 

0] 

Bkgrnd Blue: [32 239 

4 

0 

0 

0 

0 

0] 

Ll diff: 

0.0351314 





n: 

0.62253 






e : 

3.13577 






pdet: 0.184487 
p(5.0): 0.0677602 








Table 6. Detection probability comparison table and program output for Screen 
Shot 25 
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Screen Shot 25, depleted in Figure 21, was another image where our algorithm 
failed to replieate the results found in the experiment. From the contrast calculations one 
ean see that there is a very low level of eontrast between the target and the baekground. 
Why was the deteetion rate so high in the experiment? There are several possible 
reasons. First, the viable hiding places in this scene are limited. One can cycle though 
these places quickly and, thus, take a closer look at eaeh. Also, the target appears near 
the edge of the scene. We suspect that this increases the likelihood of deteetion. 

Of additional interest is the large variance in the deteetion times of the 
experimental subjects. This indicates that Sereen Shot 25 was, in fact, a difficult scene 
for the subjects. Flowever, the subjects were told that there was likely a target in the 
seene and so they were less likely to give up. This undoubtedly adds to the high 
deteetion percentages. 
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7. 


Screen Shot 37 



Figure 24. Screen Shot 37 



Figure 25. Screen Shot 37 with false coloring 
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Time to sucessful detection 



Seconds to Detection 


Figure 26. Distribution of successful experimental detection times for Screen Shot 37 


Source 

Time-independent Probability 
of Detection 

Probability of Detection 

within 5 Seconds 

Experiment 

0.5000 

0.1429 

ACQUIRE-like algorithm 

1.0000 

0.9472 

Color-modified ACQUIRE 

1.0000 

0.9472 

Color Histograms 

0.9999 

0.9472 


Processing Frames . . . 

Using the ACQUIRE-like algorithm 

Ave. Target Pix Val: 63.8151 Ave. Bckgrd Pix Val: 39.9457 

Contrast Value: 1.55216 

n: 19.3242 

e: 16.2269 

pdet: 1 

p (5.0) : 0.947196 

Using the color-aware ACQUIRE-like algorithm 



Ave. 

Target 

Red Val: 

66.5613 

Ave. 

Bckgrd 

Red Val: 

42.837 



Ave. 

Target 

Grn Val: 

66.8774 

Ave. 

Bckgrd 

Grn Val: 

44.25 



Ave. 

Target 

Blu Val: 

58.0065 

Ave. 

Bckgrd 

Blu Val: 

32.75 



Contrast Value: 

1.6119 







n: 



20.0679 







e: 



16.7476 






pdet: 

L 









p(5.0) 

: 0.947196 








Using color 

histograms 







Target 

Red: 

[6 

51 

94 

4 

0 

0 

0 

0] 

Bkgrnd 

Red: 

[30 

44 

15 

3 

0 

0 

0 

0] 

Target 

Green 

: [5 

54 

91 

5 

0 

0 

0 

0] 

Bkgrnd 

Green 

: [23 

52 

13 

4 

0 

0 

0 

0] 

Target 

Blue: 

[12 

88 

54 

1 

0 

0 

0 

0] 

Bkgrnd 

Blue : 

[63 

20 

8 

1 

0 

0 

0 

0] 


LI diff: 


0.45029 







n: 



5.60612 







e : 



6.62429 







pdet: 0.999989 
p (5.0) : 0.947186 


Table 7. Detection probability comparison table and program output for Screen 
Shot 37 
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Screen Shot 37, depieted in figure 24, is an example of a seene where the eontrast 
between the target and the baekground is substantial, yet the experimental detection rate 
was low eompared to our algorithms. Note that the edges of the target are obstrueted 
from view, in effeet masking the human-like aspeets of the target and, thus, making the 
target difficult to detect, despite the eontrast. The amount of elutter in the scene adds to 
the confusion. Additionally, colors similar to the targets colors appear elsewhere in the 
scene. Our ACQUIRE-like algorithms take none of these issues into account. 

D. CONCLUSIONS AND SUGGESTED FUTURE WORK 

Little is known about how well perception in 3-D virtual environments, either by 
algorithms or by humans looking at sereens or with head mounted displays (HMDs), 
aetually eorresponds to perception in the real world. There is simply no data on this 
subjeet that we are aware of Consider if you eame across the seene in Figure 13 in real 
life. It is likely that you would deteet the target in this situation, probably rather quiekly. 
There are several reasons for this. First are the benefits that come from 3-dimensional 
eyesight. The target would stand out from the wall, unless he was in deep shadow at 
whieh the eye becomes less effeetive. Second, we know that movement plays a large 
role in target detection. So, if the target moved at all we would likely detect him as well. 
Reece and Wirthlan address motion, speeifically, making the probability of detection 
higher for a moving target, but there are still a number of issues to address in this line of 
researeh, including that, in some cases motion might mask deteetion. Consider a soldier 
moving in a forest with a myriad of wind blown leaves or a soldier moving through 
swirling smoke. Improved algorithms will have a positive impaet on gaming and military 
training simulation emersion and improve the validity of military simulations. 

We have only seratched the surfaee of what ean be done with image inputs to 
pereeption algorithms. However, we have shown that sueh images can be used as inputs 
for these algorithms and that such algorithms can outperform line-of-sight traees. The 
fields of computer vision and image proeessing undoubtedly have a great deal to add to 
this line of research. Further development of algorithms using these techniques is 
eneouraged. 
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